A long time ago, when I was first introduced to email, I was using the
Mail program from Unix. I quickly converted to Elm, then Mutt, which
were both better in terms of interface. Then I found out about Gnus,
and I wouldn't dream of letting it go now. However, Gnus has started
showing its age several times, and several times have I needed to
upgrade the way I was using it: first because I needed to sort and
split email, then because I took the sorting out of Gnus and into
Procmail for more advanced filtering (including spam filtering), then
because I switched to storing the emails on an IMAP server so I could
read them remotely from several computers. My setup as of a few days
ago was functional, but since I have grown over time to splitting
emails into several hundred folders, checking for new messages was
becoming more and more boring.
So it's time to jump in with all the cool kids and switch to a modern
solution: still Gnus of course, but with Dovecot, OfflineIMAP for
synchronisation, and let's add email searches into the mix while we're
at it. My web searches didn't turn up a simple step-by-step HOWTO,
but I assembled bits from different places, and here's my attempt at
documenting my new setup.
Goals
- Gnus
- Offline operation
- Searching through emails
- and make it fast!
Assumptions
- Email gets delivered (by SMTP or fetchmail or whatever) to a remote
server, which can be accessed through IMAP.
- The client is running some sort of standard Unix-like system; Debian
GNU/Linux for me, but it should also work with a BSD or Solaris or
something else.
- There is enough space on the client to have a copy of the whole
email store (or at least of the folders you want to use with this
setup Gnus can of course access the others through the slower IMAP
server).
Dovecot setup
We'll use
Dovecot as a local IMAP server.
And since we're lazy, we'll access it over a pipe, and dispense with
the network part.
- Install Dovecot (
aptitude install dovecot-imapd on Debian systems
and related).
- Disable its automated running unless you need it for other purposes;
on Debian, set
ENABLED=0 in /etc/default/dovecot.
- Decide on where you'll store your emails locally; typical place
would be
$HOME/Maildir.
- That's it!
OfflineIMAP setup
OfflineIMAP is
basically an optimised two-way synchronisation mechanism between two
email repositories . We'll use it in IMAP-to-IMAP mode.
aptitude install offlineimap, or similar.- You'll need a
~/.offlineimaprc file, with contents based on the
following example:
[general]
accounts = MyAccount
pythonfile = .offlineimap.py
[Account MyAccount]
localrepository = LocalIMAP
remoterepository = RemoteIMAP
# autorefresh = 5
# postsynchook = notmuch new
[Repository LocalIMAP]
type = IMAP
preauthtunnel = MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap
holdconnectionopen = yes
[Repository RemoteIMAP]
type = IMAP
remotehost = mail.example.com
remoteuser = jsmith
remotepass = swordfish
ssl = yes
nametrans = lambda name: re.sub('^INBOX.', '', name)
# folderfilter = lambda name: name in [ 'INBOX.important', 'INBOX.work' ]
# folderfilter = lambda name: not (name in [ 'INBOX.spam', 'INBOX.commits' ])
# holdconnectionopen = yes
maxconnections = 3
# foldersort = lld_cmp
- You will of course need to replace the
remotehost, remoteuser
and remotepass values with your own. If you want to synchronize
only a subset of the IMAP folders, uncomment and adapt one of the
folderfilter lines; the lambda can be any Python code you like, I
guess.
- Run
offlineimap. See the emails coming in and being replicated
to your local store.
- Run
MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap. You should
get * PREAUTH [CAPABILITY ... STATUS] Logged in as your-login.
Type * LIST "" * in there. You should see a list of all the
folders.
- The
.offlineimaprc mentions a .offlineimap.py, which is where
we're going to store some additional code used by OfflineIMAP.
Here's a sample:
# Propagate gnus-expire flag
from offlineimap import imaputil
def lld_flagsimap2maildir(flagstring):
flagmap = '\\seen': 'S',
'\\answered': 'R',
'\\flagged': 'F',
'\\deleted': 'T',
'\\draft': 'D',
'gnus-expire': 'E'
retval = []
imapflaglist = [x.lower() for x in flagstring[1:-1].split()]
for imapflag in imapflaglist:
if flagmap.has_key(imapflag):
retval.append(flagmap[imapflag])
retval.sort()
return retval
def lld_flagsmaildir2imap(list):
flagmap = 'S': '\\Seen',
'R': '\\Answered',
'F': '\\Flagged',
'T': '\\Deleted',
'D': '\\Draft',
'E': 'gnus-expire'
retval = []
for mdflag in list:
if flagmap.has_key(mdflag):
retval.append(flagmap[mdflag])
retval.sort()
return '(' + ' '.join(retval) + ')'
imaputil.flagsmaildir2imap = lld_flagsmaildir2imap
imaputil.flagsimap2maildir = lld_flagsimap2maildir
# Grab some folders first, and archives later
high = ['^important$', '^work$']
low = ['^archives', '^spam$']
import re
def lld_cmp(x, y):
for r in high:
xm = re.search (r, x)
ym = re.search (r, y)
if xm and ym:
return cmp(x, y)
elif xm:
return -1
elif ym:
return +1
for r in low:
xm = re.search (r, x)
ym = re.search (r, y)
if xm and ym:
return cmp(x, y)
elif xm:
return +1
elif ym:
return -1
return cmp(x, y)
The first part of this file adds a new flag that OfflineIMAP will
propagate back and forth. By default, only the standard IMAP flags
are propagated; we also want to synchronize the
gnus-expire flag
that Gnus uses to mark expirable articles. It's a hack, but it works
for now (maybe someday OfflineIMAP will propagate all the flags it
finds?).
The second part of that file can be dispensed with (and won't be used
unless the
foldersort option is uncommented in
.offlineimaprc):
it's only there to ensure that some important folders are propagated
first, and some others go last. I don't know exactly how they are
sorted by default, but I'd like the most important ones to come first,
so I can start reading them while the archives and the spam are still
being fetched.
Gnus configuration
- You need to set up a select method for the local IMAP server. The
method will be
nnimap, the address can be whatever you like (it's
just a name anyway), and you only need two options: nnimap-stream,
set to shell, and imap-shell-program, which you set to
"MAIL=maildir:$HOME/Maildir /usr/lib/dovecot/imap".
- Then browse the server, subscribe to the folders, and so on.
- From time to time, run
offlineimap from a terminal, so you get the
new messages, delete the old ones, and so on.
- What's that? You hate running stuff by hand? So do I. To run
OfflineIMAP in a loop, uncomment the
autorefresh option in
.offlineimaprc. The value is the duration (in minutes) between
two runs.
- Even starting OfflineIMAP is too much work?
offlineimap.el to
the rescue! Grab it, put it where Emacs will find it, and add the
following to your .emacs:
(require 'offlineimap)
(add-hook 'gnus-before-startup-hook 'offlineimap)
Bonus: email searches
The simple way:
(require 'nnir)
This goes in your
.gnus. Then your group buffer will get a new
command. Mark some folders with
#, then
M-x
gnus-group-make-nnir-group (or use the
G G shortcut), and type in a
set of keywords. This search is performed by the IMAP server
(Dovecot), which may or may not be very efficient, especially if you
select many folders.
Bonus+: email searches, faster
The
real cool kids use Notmuch nowadays, at least for email indexing
and searching. It's fast, it allows complex queries, and it's
generally cool. The downside is that it uses up quite some disk space
for its indices, in addition to the actual emails. For that reason
I'll keep it to my main computer, and I'll stick to
nnir on my
laptop (which has the same setup apart from that).
aptitude install notmuch, yada yada.notmuch setup, tell it where you stored your Maildir. If you're
only going to use Notmuch for searches, I suggest setting tags= to
an empty value in .notmuch-config afterwards. We don't want
Notmuch to intrude.notmuch new will get it indexing the messages you already have.- Of course, being lazy is being cool, so let's uncomment the
postsynchook line from .offlineimaprc.
- Now add
(require 'notmuch) to your .gnus. Also (define-key
gnus-group-mode-map "GG" 'notmuch-search), for the shortcut.
- But seeing just the message that matches is not enough, sometimes we
want the whole thread. Here's a snippet of Lisp for your
.gnus,
based on
Tassilo Horn's configuration.
Do a Notmuch search, enter one of the results, type C-c C-c,
you'll get transported to the folder where that message was, with
the context. Note this requires code from org-mode, so you might
need to install that.
(require 'notmuch)
(add-hook 'gnus-group-mode-hook 'lld-notmuch-shortcut)
(require 'org-gnus)
(defun lld-notmuch-shortcut ()
(define-key gnus-group-mode-map "GG" 'notmuch-search)
)
(defun lld-notmuch-file-to-group (file)
"Calculate the Gnus group name from the given file name.
"
(let ((group (file-name-directory (directory-file-name (file-name-directory file)))))
(setq group (replace-regexp-in-string ".*/Maildir/" "nnimap+local:" group))
(setq group (replace-regexp-in-string "/$" "" group))
(if (string-match ":$" group)
(concat group "INBOX")
(replace-regexp-in-string ":\\." ":" group))))
(defun lld-notmuch-goto-message-in-gnus ()
"Open a summary buffer containing the current notmuch
article."
(interactive)
(let ((group (lld-notmuch-file-to-group (notmuch-show-get-filename)))
(message-id (replace-regexp-in-string
"^id:" "" (notmuch-show-get-message-id))))
(if (and group message-id)
(progn
(switch-to-buffer "*Group*")
(org-gnus-follow-link group message-id))
(message "Couldn't get relevant infos for switching to Gnus."))))
(define-key notmuch-show-mode-map (kbd "C-c C-c") 'lld-notmuch-goto-message-in-gnus)
Wrap-up
This setup can be replicated on several computers, of course. I have
it on two, and there's no reason I couldn't have more. The flags do
get propagated back and forth, including the Gnus-specific expirable
flag. Accessing the local Dovecot is
much faster than going through
the DSL to the master IMAP server, and I'm pretty convinced that
OfflineIMAP and its multi-threading is also faster than Gnus is, even
talking to the same remote server. The email searching with Notmuch
is a nice bonus, especially since they're
fast too (and this despite
my 8-year-old computer).
There are a few minor glitches. I can live with them, but I should
let you know anyway.
offlineimap.el keeps an OfflineIMAP process running (if using
autorefresh), which causes Emacs to complain about when you want
to exit.- OfflineIMAP propagates the new folders from the remote server to the
local store, but not the other way round. Folders created from Gnus
won't be propagated, so they'll only be visible on one computer.
Apart from that, I'm pretty happy with this new setup. So I hope this
documentation will be useful to others, so I can spread the happiness
around. Send your thanks to the authors of the software involved
(Gnus, Dovecot, OfflineIMAP,
offlineimap.el, Procmail, and so on).
Let's see how many years I'll keep that system!
Martin Michlmayr is a Debian developer since 2000 and I share quite a few things with him, starting with his age and involvement in the quality assurance team. He managed to be elected Debian Project Leader in 
GNOME 3 packaging
Right after the GNOME 3 release, I was eager to try it out so I 
I ve been running this code for more than 2 months and it works fine. I want to see it merged. I m ready to update my code should anything need to be changed to please Guillem. But without any feedback we re in a deadlock.
Misc dpkg work
While fixing a bug in update-alternatives (found in one of the valid reports on launchpad), I noticed that there was room for improvements in the error messages output by update-alternatives. I changed them to reuse the same strings that were already used in other parts of dpkg. The result is that there are a few strings less to translate (always a nice thing for the poor translators who have to deal with the thousands of strings that dpkg contains).
I also tried to fix some of the most cryptic error messages in dpkg (see 
I asked 
First steps as a GNOME packager
This is pretty common in free software and for once I followed the advice, I spent most of sunday helping out with GNOME 3 packaging. I have no prior experience with GNOME packaging but I m fairly proficient in Debian packaging in general so when I showed up on #debian-gnome (irc.debian.org) on sunday morning, Josselin quickly added me to the team on alioth.debian.org.
Still being a pkg-gnome rookie, I started by reading the documentation on 
Like 
Roland Mas writes